On the regularization effect of stochastic gradient descent applied to least-squares

We study the behavior of stochastic gradient descent applied to $\|Ax -b \|_2^2 \rightarrow \min$ for invertible $A \in \mathbb{R}^{n \times n}$. show that there is an explicit constant $c_{A}$ depending (mildly) on $A$ such $$ \mathbb{E} ~\left\| Ax_{k+1}-b\right\|^2_{2} \leq \left(1 + \frac{c_{A}}{\|A\|_F^2}\right) \left\|A x_k \right\|^2_{2} - \frac{2}{\|A\|_F^2} \left\|A^T A (x_k x)\right\|^2_{2}.$$ This a curious inequality: last term has one more matrix residual $u_k u$ than remaining terms: if $x_k x$ mainly comprised large singular vectors, leads quick regularization. For symmetric matrices, this inequality extension higher-order Sobolev spaces. explains (known) regularization phenomenon: energy cascade from values small smoothes.